Learning Predictive State Representations

نویسندگان

  • Satinder P. Singh
  • Michael L. Littman
  • Nicholas K. Jong
  • David Pardoe
  • Peter Stone
چکیده

We introduce the rst algorithm for learning predictive state representations (PSRs), which are a way of representing the state of a controlled dynamical system. The state representation in a PSR is a vector of predictions of tests, where tests are sequences of actions and observations said to be true if and only if all the observations occur given that all the actions are taken. The problem of nding a good PSR|one that is a suÆcient statistic for the dynamical system|can be divided into two parts: 1) discovery of a good set of tests, and 2) learning to make accurate predictions for those tests. In this paper, we present detailed empirical results using a gradient-based algorithm for addressing the second problem. Our results demonstrate several sample systems in which the algorithm learns to make correct predictions and several situations in which the algorithm is less successful. Our analysis reveals challenges that will need to be addressed in future PSR learning algorithms.

برای دانلود رایگان متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Predictive State Representations with State Space Partitioning

Predictive state representations (PSRs) are powerful methods of modeling dynamical systems by representing state through observational data. Most of the current PSR techniques focus on learning a complete PSR model from the entire state space. Consequently, the techniques are often not scalable due to the dimensional curse, which limits applications of PSR. In this paper, we propose a new PSR l...

متن کامل

Mixed Observability Predictive State Representations

Learning accurate models of agent behaviours is crucial for the purpose of controlling systems where the agents’ and environment’s dynamics are unknown. This is a challenging problem, but structural assumptions can be leveraged to tackle it effectively. In particular, many systems exhibit mixed observability, when observations of some system components are essentially perfect and noiseless, whi...

متن کامل

Using Predictive Representations to Improve Generalization in Reinforcement Learning

The predictive representations hypothesis holds that particularly good generalization will result from representing the state of the world in terms of predictions about possible future experience. This hypothesis has been a central motivation behind recent research in, for example, PSRs and TD networks. In this paper we present the first explicit investigation of this hypothesis. We show in a r...

متن کامل

Goal-Directed Online Learning of Predictive Models

We present an algorithmic approach for integrated learning and planning in predictive representations. The approach extends earlier work on predictive state representations to the case of online exploration, by allowing exploration of the domain to proceed in a goal-directed fashion and thus be more efficient. Our algorithm interleaves online learning of the models, with estimation of the value...

متن کامل

A Survey of Predictive State Representations

Predictive State Representations (PSRs) [10] are a model for a discrete-time finite action and observation stochastic systems, presented as an alternative to HMMs and POMDPs. A PSR represents the system’s state as a set of predictions of the observable outcomes of tests performed in the system. Unlike hidden variable models no latent variables are assumed or required – only observable outcomes ...

متن کامل

Schema Learning: Experience-Based Construction of Predictive Action Models

Schema learning is a way to discover probabilistic, constructivist, predictive action models (schemas) from experience. It includes methods for finding and using hidden state to make predictions more accurate. We extend the original schema mechanism [1] to handle arbitrary discrete-valued sensors, improve the original learning criteria to handle POMDP domains, and better maintain hidden state b...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2003